Mean-field games have been used as a theoretical tool to obtain an approximate Nash equilibrium for symmetric and anonymous $N$-player games in literature. However, limiting applicability, existing theoretical results assume variations of a "population generative model", which allows arbitrary modifications of the population distribution by the learning algorithm. Instead, we show that $N$ agents running policy mirror ascent converge to the Nash equilibrium of the regularized game within $\tilde{\mathcal{O}}(\varepsilon^{-2})$ samples from a single sample trajectory without a population generative model, up to a standard $\mathcal{O}(\frac{1}{\sqrt{N}})$ error due to the mean field. Taking a divergent approach from literature, instead of working with the best-response map we first show that a policy mirror ascent map can be used to construct a contractive operator having the Nash equilibrium as its fixed point. Next, we prove that conditional TD-learning in $N$-agent games can learn value functions within $\tilde{\mathcal{O}}(\varepsilon^{-2})$ time steps. These results allow proving sample complexity guarantees in the oracle-free setting by only relying on a sample path from the $N$ agent simulator. Furthermore, we demonstrate that our methodology allows for independent learning by $N$ agents with finite sample guarantees.

Given a particular embodiment, we propose a novel method (C3PO) that learns policies able to achieve any arbitrary position and pose. Such a policy would allow for easier control, and would be re-useable as a key building block for downstream tasks. The method is two-fold: First, we introduce a novel exploration algorithm that optimizes for uniform coverage, is able to discover a set of achievable states, and investigates its abilities in attaining both high coverage, and hard-to-discover states; Second, we leverage this set of achievable states as training data for a universal goal-achievement policy, a goal-based SAC variant. We demonstrate the trained policy's performance in achieving a large number of novel states. Finally, we showcase the influence of massive unsupervised training of a goal-achievement policy with state-of-the-art pose-based control of the Hopper, Walker, Halfcheetah, Humanoid and Ant embodiments.

当今许多大型系统的设计，从交通路由环境到智能电网，都依赖游戏理论平衡概念。但是，随着$ n $玩家游戏的大小通常会随着$ n $而成倍增长，标准游戏理论分析实际上是不可行的。最近的方法通过考虑平均场游戏，匿名$ n $玩家游戏的近似值，在这种限制中，玩家的数量是无限的，而人口的状态分布，而不是每个单独的球员的状态，是兴趣。然而，迄今为止研究最多的平均场平衡的平均场nash平衡的实际可计算性通常取决于有益的非一般结构特性，例如单调性或收缩性能，这是已知的算法收敛所必需的。在这项工作中，我们通过开发均值相关和与粗相关的平衡的概念来研究平均场比赛的替代途径。我们证明，可以使用三种经典算法在\ emph {ash All Games}中有效地学习它们，而无需对游戏结构进行任何其他假设。此外，我们在文献中已经建立了对应关系，从而获得了平均场 - $ n $玩家过渡的最佳范围，并经验证明了这些算法在简单游戏中的收敛性。

具有很多玩家的非合作和合作游戏具有许多应用程序，但是当玩家数量增加时，通常仍然很棘手。由Lasry和Lions以及Huang，Caines和Malham \'E引入的，平均野外运动会（MFGS）依靠平均场外近似值，以使玩家数量可以成长为无穷大。解决这些游戏的传统方法通常依赖于以完全了解模型的了解来求解部分或随机微分方程。最近，增强学习（RL）似乎有望解决复杂问题。通过组合MFGS和RL，我们希望在人口规模和环境复杂性方面能够大规模解决游戏。在这项调查中，我们回顾了有关学习MFG中NASH均衡的最新文献。我们首先确定最常见的设置（静态，固定和进化）。然后，我们为经典迭代方法（基于最佳响应计算或策略评估）提供了一个通用框架，以确切的方式解决MFG。在这些算法和与马尔可夫决策过程的联系的基础上，我们解释了如何使用RL以无模型的方式学习MFG解决方案。最后，我们在基准问题上介绍了数值插图，并以某些视角得出结论。

已经引入了平均野外游戏（MFG），以有效地近似战略代理人。最近，MFG中学习平衡的问题已经获得了动力，尤其是使用无模型增强学习（RL）方法。使用RL进一步扩展的一个限制因素是，解决MFG的现有算法需要混合近似数量的策略或$ Q $价值。在非线性函数近似的情况下，这远非微不足道的属性，例如，例如神经网络。我们建议解决这一缺点的两种方法。第一个从历史数据蒸馏到神经网络的混合策略，将其应用于虚拟游戏算法。第二种是基于正规化的在线混合方法，不需要记忆历史数据或以前的估计。它用于扩展在线镜下降。我们从数值上证明，这些方法有效地可以使用深RL算法来求解各种MFG。此外，我们表明这些方法的表现优于文献中的SOTA基准。

已经提出了几种算法，以非均匀地对深钢筋学习（RL）剂的重播缓冲液进行采样，以加速学习，但是几乎没有提供这些抽样方案的理论基础。除其他外，优先的经验重播似乎是一种超级参数敏感的启发式，尽管它可以提供良好的性能。在这项工作中，我们将重播缓冲液抽样问题视为估算梯度的重要性采样。这允许得出理论上最佳的采样分布，从而获得最佳的理论收敛速度。详细阐述了理想抽样方案的知识，我们展示了优先经验重播的新理论基础。最佳采样分布非常棘手，我们进行了几个近似值，可在实践中提供良好的结果，并介绍Laber（大批次经验重播），这是一种易于编码和有效的方法来抽样重播缓冲区。与其他优先级方案相比，Laber可以与深层Q-NETWORKS，分布RL代理或参与者 - 批判性方法结合使用，在各种Atari游戏和Pybullet环境中，可以提高性能。

常见的策略梯度方法依赖于代理函数序列的最大化。近年来，已经提出了许多这样的代理功能，大多数没有强烈的理论担保，导致TRPO，PPO或MPO等算法。我们而不是设计另一个代理函数，而是根据功能镜中的函数提出一般框架（FMA-PG），这导致了整个代理功能。我们构建了使策略改进保证能够担保的代理功能，这是由最现有的代理职能共享的属性。至关重要，无论政策参数化的选择如何，这些保证都会持有。此外，FMA-PG的特定实例化恢复了重要的实施启发式（例如，使用前向VS反向KL发散），导致TRPO的变体具有额外的理想性质。通过对简单强盗问题的实验，我们评估FMA-PG实例化的算法。拟议的框架还提出了一种改进的PPO变体，其鲁棒性和效率我们在Mujoco套件上证明。

政策评估算法对于加强学习是必不可少的，因为他们预测了政策的表现。然而，在这种预测问题中有两个长期存在的问题，需要解决：违规稳定性和持续政策效率。已知传统的时间差（TD）算法在策略设置中执行非常好，但不是障碍稳定。另一方面，梯度TD和强调TD算法是截止策略稳定性的，但不是策略的高效。本文介绍了通过使用倾斜投影方法的非禁止稳定性和持续策略的新算法。各个领域的经验实验结果验证了所提出的方法的有效性。

Foundation models are redefining how AI systems are built. Practitioners now follow a standard procedure to build their machine learning solutions: download a copy of a foundation model, and fine-tune it using some in-house data about the target task of interest. Consequently, the Internet is swarmed by a handful of foundation models fine-tuned on many diverse tasks. Yet, these individual fine-tunings often lack strong generalization and exist in isolation without benefiting from each other. In our opinion, this is a missed opportunity, as these specialized models contain diverse features. Based on this insight, we propose model recycling, a simple strategy that leverages multiple fine-tunings of the same foundation model on diverse auxiliary tasks, and repurposes them as rich and diverse initializations for the target task. Specifically, model recycling fine-tunes in parallel each specialized model on the target task, and then averages the weights of all target fine-tunings into a final model. Empirically, we show that model recycling maximizes model diversity by benefiting from diverse auxiliary tasks, and achieves a new state of the art on the reference DomainBed benchmark for out-of-distribution generalization. Looking forward, model recycling is a contribution to the emerging paradigm of updatable machine learning where, akin to open-source software development, the community collaborates to incrementally and reliably update machine learning models.

One of the major challenges of machine translation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as an image. However, recent work in multimodal MT (MMT) has shown that obtaining improvements from images is challenging, limited not only by the difficulty of building effective cross-modal representations but also by the lack of specific evaluation and training data. We present a new MMT approach based on a strong text-only MT model, which uses neural adapters and a novel guided self-attention mechanism and which is jointly trained on both visual masking and MMT. We also release CoMMuTE, a Contrastive Multilingual Multimodal Translation Evaluation dataset, composed of ambiguous sentences and their possible translations, accompanied by disambiguating images corresponding to each translation. Our approach obtains competitive results over strong text-only models on standard English-to-French benchmarks and outperforms these baselines and state-of-the-art MMT systems with a large margin on our contrastive test set.